AO-AOSO  093 
UNCLASSIFIED 


TEXAS  UNIV  AT  AUSTIN  CENTER  FOR  CYBERNETIC  STUDIES  F/B  8/1 

AU0ITIN8  AND  ACCOUNTING  FOR  PROGRAM  EFFICIENCY  ANO  MANAGEMENT  E— ETC(U) 
AU#  7f  A  CHARNESt  M  ■  COOPER  N0001A-7S-C-0G16 

CCS-3W  NL 


IDAU8U453 


(7 )  — — — 7 

I  Research  /iiepa»t,\pCS-»343  / 

a'uotting  And  accounting 

\  FOR  PROGRAM. EFFICIENCY  AND 

/  ma'Magement'efficiency  in 

'^OT-FOR -PROFIT  ENTITIES. 


by 

A./ Cham 
^  W.W./Coop 


£  o 


:  y  r 


Y  Augwt  1*79 

' ' . -  •  fe/- 


7  //;•  7 


d-  d  / 


♦The  University  of  Texas  at  Austin 

♦♦Harvard  University  Graduate  School  of  Business 


This  paper  represents  the  draft  of  a  paper  to  be  presented  at  the 
August  24,  1979,  plenary  session  of  the  American  Accounting  Association 
Honolulu  Convention,  It  was  partly  supported  by  NSF  Grant  No.  SOC76- 
15876,  "Collaborative  Research  on  the  Analytical  Capabilities  of  a  Goals 
Accounting  System."  It  is  also  part  of  the  program  of  the  Navy  Personnel 
Research  and  Development  Center  and  is  partly  supported  by  Projects 
NR047-071  and  021,  ONR  Contracts  N00014-75-C-0616  and  0569  with  the 
Center  for  Cybernetic  Studies,  The  University  of  Texas  at  Austin,  and 
ONR  Contract  N00014-75-C-0932  at  Carnegie-Mellon  University  School  of 
Urban  and  Public  Affairs.  Reproduction  in  whole  or  in  part  is  permitted 
for  any  purpose  of  the  U.S.  Government. 


CENTER  FOR  CYBERNETIC  STUDIES 

A.  Charnes,  Director 
Business-Economics  Building,  203E 
The  University  of  Texas  at  Austin 
Austin,  TX  78712 
(512)  471-1821  ,  / 


I/O 6  lC/'i 


COMPREHENSIVE  AUDIT,  AN  INTRODUCTION 


A  plenary  address  in  the  same  session  with  U.S.  Comptroller  General 
Elmer  Staats  provides  an  opportunity  to  relate  this  presentation  to  the 
new  kinds  of  audits  which  his  organization,  the  U.S.  General  Accounting  Office, 
has  pioneered.  This  type  of  audit,  which  we  have  elsewhere  referred  to  as  "com¬ 
prehensive  audit,"  is  also  being  developed  in  other  governments  (both  foreign 

and  domes tic)^ and  in  the  internal  audit  practices  of  some  of  the  large  multina- 

2/ 

tional  corporations.—  Nevertheless  the  U.S.  General  Accounting  Office,  under  Dr. 

3/ 

Staats,  must  be  accorded  the  position  of  leader  in  these  important  developments- 
and  it  is  their  terminology  and  concepts  which  we  shall  adapt  as  a  guide  in  the 
discussion  that  follows. 

The  term  "audit"  is  often  viewed,  especially  by  accountants,  as  synonymous 
with  the  CPA  function  of  attesting  to  financial  statements  and  related  represen¬ 
tations  by  management.  The  types  of  audit  we  are  considering,  although  broader 
in  scope,  do  continue  the  CPA  tradition  of  servicing  "third  party"  needs  for 
information  as  one  of  its  major  functions  —  along  with  corollary  responsi¬ 
bilities  for  objectivity  and  professional  canons  of  care  and  validation.  In 
operational  terras  this  means  that  conditions  such  as  auditor  independence  and 
attention  to  conditions  of  fairness  (or  balance) and  evidentiary  documentation 
must  attend  all  aspects  of  the  appraisals  that  are  undertaken  in  these  comprehen¬ 
sive  audits. 

The  term  "comprehensive"  seems  appropriate  because  this  type  of 
audit  extends  the  process  of  objective  appraisal  to  all  aspects  of  manage¬ 
ment.  By  this  we  mean  that  the  auditor  assumes  responsibility  for  the 
aspect  of  management  that  is  designated  for  audit  as  well  as  the  way  the 

1Vlde  [32]  and  [33]  by  Dr.  Staats  and  the  report  of  the  Auditor  General  of  Canada  [17]. 
Other  developments  are  also  examined  in  [20]  and  [21].  Se  also  [22]. 

^See  the  discussion  in  [11].  See  also  [10], 

3Cf.  [12.1M12.3J.  See  also  118]  and  [33]. 
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audit  ia  conducted.  Under  comprehensive  audit,  the  auditor  does  not  stop 
short  with  the  attest  function.  Instead,  he  also  assumes  responsibility 
for  the  report  that  is  rendered,  as  well  as  the  third  party  groups  to  be 
serviced  by  the  audit  process.  Finally,  instead  of  restricting  audit 
examinations  to  financial  transactions  (and  their  representations  in  finan¬ 
cial  reports),  such  comprehensive  audits  extend  to  examination  and  appraisal 
of  any  of  the  following  aspects  of  management  and  organization  behavior: 

Table  1 


1.  Propriety  of 

a.  Objectives  pursued 

b.  Methods  used 

2.  Effectiveness  in 

a.  Stating  objectives 

b.  Attaining  objectives 

3.  Efficiency  of  performance  as  measured  by 

a.  Benefits  received 

b.  Resources  utilized. 

Some  or  all  of  these  aspects  of  "comprehensive  audit,"  as  we 
have  already  observed,  are  to  be  found  in  well-developed  forms  in  certain 
types  of  governmental  and  internal  corporate  audit  activities.  They  are 
also  to  be  found  in  areas  as  widely  dispersed  as  the  audit  of  CPA  firms 
by  other  CPAs^in  what  are  called  "peer  reviews"  and  extend  also  to  peer  re¬ 
view  processes  that  are  now  being  used  to  audit  and  appraise  the  practices  of 

2 

physicians  in  various  U.S.  hospitals,  and  include  the  efficiency  audits  that 

are  now  being  ordered  by  public  utility  commissions  for  the  utilities  under 
3/ 

their  jurisdiction.— 

^See  [25].  See  also  pp.  34ff.  in  Appendix  E  to  [34]  for  disccussion  of  the  need 
for  public  disclosure  of  the  results  of  such  peer  reviews  in  order  to  service  3rd 
party  needs  for  information  and  evaluation. 

2 

E.g.,  by  so-called  PSRO's  Professional  Standards  Review  Organizations).  See 
Govindarajan  |10]  for  further  discussion  in  a  study  that  explicitly  relates  these 
PSRO  procedures  to  the  audit  practices  we  are  describing. 

3see  San  Miguel  [29]  and  [30]. 


In  search  of  an  enhanced  accountability,  these  extensions  of  the 
audit  function  are  also  being  accompanied  by  alterations  in  corporate  and 
government  agency  governance  structures.  Witness,  for  instance,  the  cur¬ 
rent  attention  to  reforms  such  as  the  new  Inspector  General  legislation^  and 

the  attention  to  outside  directors  and  audit  committees  in  private  corpora- 
2/ 

tions—  These  all  represent  attempts  at  evolution  toward  new  forms  of  social 

and  management  controls  which  can  (and  should)  be  aided  by  the  kinds  of  / 

research  that  academic  accountants  (and  others)  can  supply. 


*See  the  discussion  in  the  paper  (32]  by  Dr.  Staats. 

2/ 

—  A  report  of  the  SEC’s  views  on  the  current  state  of  these  developments  may 
be  found  on  pp.  6ff.  in  Appendix  D  of  [34]. 
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2.  DECISION-MAKING  EFFICIENCY 

It  is  one  purpose  of  the  present  paper  to  try  to  effect  such  a 
contribution.  In  particular  we  want  to  introduce  a  new  way  to  evaluate 
the  efficiency  of  management  in  the  not-for-profit  entities  that  consti¬ 
tute  such  a  large  (and  growing)  part  of  our  economy.  We  want  to  do  this, 
moreover,  in  a  manner  that  opens  the  possibility  of  distinguishing  between 
programs,  policies,  organization  designs,  and  related  assignments  of 
decision-making  responsibilities,  etc.,  in  order  to  help  ascertain  how  these 
may  serve  to  enhance  or  impede  decision-making  efficiency. 

Referring  to  Table  1,  above,  we  might  specifically  say  that  we 
are  directing  our  attention  to  the  synthesis  of  a  single  number  summary 

to  help  measure  the  efficiency  of  not-for-profit  entities. 

This  is  to  be  done  in  a  way  that  takes  account  of  the  fact  that  the  activi¬ 
ties  of  most  such  entities  involve  multiple  outputs  and  multiple  inputs 
which  are  often  remote  from  any  markets.  We  therefore  want  to  avoid 
the  need  for  a  priori  weights  such  as  imputed  market  prices,  etc.,  in  our 
measure  of  efficiency.  We  seek  instead  to  synthesize  a  method  for  objectively 
determining  the  weights  we  shall  be  using.  Then  we  shall  supply  inter¬ 
pretations  that  will  lend  meaning  to  our  results  and  also  open  avenues  for 
uses  that  extend  beyond  the  particular  applications  that  we  shall  be 
exploring. 

Ways  in  which  our  approach  might  be  related  to  audits  and  other 
aspects  of  accounting  will  also  be  indicated  at  appropriate  places  in  the 
sections  that  follow.  To  conclude  the  present  section,  however,  we  might 
first  observe  that  the  private  enterprise  sector  already  has  a  measure,  viz.,  net 
profit, which  simultaneously  accommodates  both  the  effectiveness  and  efficiency 


The  measure  we 


-  5  - 

components  which  are  listed  in  Table  1.—^ 
shall  be  exploring  is  more  limited  in  that  it  is  restricted  to 

the  efficiency  dimension  only.  For  public  programs  and  not-for-profit 
entities,  this  seems  to  be  the  best  way  to  proceed.  Note,  for  instance, 
that  it  is  acceptable  private  enterprise  practice  for  management  to  invest 
in  new  undertakings  which  are  far  removed  from  past  and  present  activities. 
Such  a  diversion  of  funds  to  programs  which  are  far  removed  from  present  and 
past  activities  would  not,  in  general,  be  available  to  the  management  of  a 
government  agency,  say,  without  first  securing  legislative  and/or  other 
authorizations  -  perhaps  including  even  public  referenda  —  and  such  freedom 
to  specify  directions  as  well  as  the  magnitude  of  the  efforts  to  be  undertaken 
are  components  that  would  need  to  be  evaluated  in  terms  of  "effectiveness"  as 
well  as  "efficiency." 


^Note  that  "propriety,”  in  the  sense  of  Table  1,  is  not  necessarily  acsommodated-either  by 
profit  or  the  efficiency  measures  we  shall  be  examining. 
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A  MODEL  FOR  EFFICIENCY :  HYPOTHEfAL  EXAMPLE- 

The  data  in  Table  2,  below,  will  help  to  fix  ideas  in  the  developments 
pointed  toward  the  efficiency  measure  we  shall  be  examining.  Here  we  are  suppos¬ 
ing  a  hypothetical  situation  in  which  three  Decision-Making  Units^"  DMUj,DMl^,  and  DMU^ 
—  each  use  two  inputs  in  amounts  x^  and  to  produce  a  single  output  in  amount  y.  Indeed, 
we  may  imagine  the  data  in  the  table  as  referring  to  the  amounts  of  the  two  inputs  utilized 
per  unit  output  so  that  we  can  then  have  the  same  one  unit  output  reference  for  evaluating 
the  efficiency  of  each  DMU. 

Table  2 

An  Illustration  of  DMU  Efficiency 


\  DMU 

^\No. 

i 

2 

3 

Input 

X1 

2 

3 

4 

x2 

_ 

A 

L 

2 

1 

Assuming  that  all  inputs  as  well  as  the  one  output  have  some 
"value,"  we  may  note  that  DMl^  is  not  as  efficient  as  DMU^.  Observe, 
for  example,  that  DMU^  has  produced  the  same  amount  of  output  (one  unit) 
with  one  unit  less  of  the  first  input  (i.e.,  we  have  =  2  vs.  x^  =  3) 
and  no  more  of  the  second  input. 

Of  course,  the  above  characterization  is  only  qualitative. 

That  is,  it  produces  only  a  classification  of  "efficient  vs.  non-efficient" 
whereas  what  we  are  seeking  is  a  scalar  measure  of  efficiency  which  can 
be  accorded  operational  significance. 


—  This  and  subsequent  parts  of  the  present  paper  draw  heavily  on  work  done  in 
collaboration  with  E.  Rhodes  as  reported  in  [8].  See  also  [  6]  and  [9], 

2/We  Use  the  term  "Decision  Making  Units"  by  virtue  of  the  fact  that  we  are  concerned 
"Sch°°ls » "  "Hospitals,"  Government  Agencies','  etc.  and  hence  want  to 
«3if2I5*fr?Sre?5SndL,cS;.rf^h  ter“  as  etc.  See  [91 


To  obtain  the  latter  result  we  now  utilize  all  of  the  data  in 
Table  2  in  order  to  obtain  the  rating  that  is  implied  in  the  following 
model : 

lu 


max.  h  = 


o  3Vj^  +  2v2 


subject  to: 


1  > 


lu 


-  2vl  +  2v2 


(1) 


1  > 


lu 


-  3v^  +  2v2 


1  > 


lu 


-  4v^  +  lv2 


u,  v^  v2  >  0. 

Observe  that  all  three  DMUs  are  represented  in  the  constraints  in  a  manner 
which  guarantees  that  all  will  have  efficiency  ratio  values  that  lie  between 
0  and  1.  Hence  DMU2  ,  which  is  singled  out  for  evaluation  in  the  functional  to 
maximized  in  (1),  must  also  have  this  same  characteristic. 

In  conformance  with  the  measures  that  are  customary  in  both  the 
natural  and  social  sciences,  we  have  synthesized  our  model  so  that  the 
efficiency  ratings  are  in  the  form  of  ratios  that  are  limited  to  a  maximum 
value  of  100%.  Furthermore,  the  resultant  rating  can  be  shown  to  be  scale 
independent,—^  which  is  to  say  that  each  input  and  each  output  can  be  measured 
in  any  units  that  are  convenient,  without  altering  the  results,  provided 
these  same  units  are  utilized  for  every  DMU. 


^See  (6  ]  and  [26]  for  further  discussion  of  these  and  other  invariance 
properties  of  these  efficiency  measures. 
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It  should  be  observed  that  our  efficiency  rating  is  a  relative 
one.  That  is,  the  rating  of  DMli^  in  the  above  case  is  obtained  by  refer¬ 
ence  to  the  attainments  only  of  the  other  DMUs  -  viz. ,  DMU^  and  DMU^  - 
with  which  it  is  being  compared.  This  is  the  way  we  shall  proceed  although, 
in  passing,  we  might  also  remark  that  comparisons  obtained  from  engineering 
or  other  estimates  of  efficiency  may  also  be  included  when  they  are  avail¬ 
able.  Indeed,  it  can  be  shown  that  all  of  the  customary  engineer^ 

(and  natural  science)  definitions  of  ef f iciency'^can  be  accommodated  nd 
extended)  by  reference  to  the  general  form  of  the  model  that  is  represented 
in  (1). 

Although  the  model  represented  in  (1)  is  non-linear ,  it  can  also  be 
shown  to  have  a  wealth  of  properties  which  include  a  "fractional  program¬ 
ming"  equivalent  that  can,  in  turn,  be  replaced  by  an  ordinary  linear 

2 

programming  problem.  Thus,  by  using  this  route  we  may  apply  the  simplex 
method  of  ordinary  linear  programming  to  the  data  of  Table  2  in  order  to 
obtain 

(2.1)  max.  h  -  h  =6/7 

o  o 

along  with  the  optimal  weights 

(2-2)  u  =  1,  v^  -  1/6  and  v^  =  1/3. 

Substituting  these  values  in  (1)  we  obtain 


See  [9]  and  [26]  for  further  discussion  of  the  underlying  definitions  and  the  way 
these  relate  to  ideas  like  Pareto  efficiency  in  economics.  See  [6]  for  further 
discussion  in  relation  to  engineering  definitions  of  efficiency  and  a  demonstra¬ 
tion  that  the  measure  we  are  using  does  not  depend  on  the  measurement  units  used. 

» 

See  [9]  and  [26].  This  means  that  we  also  have  the  duality  theory  of  fractional 
programming  available  for  use  in  a  variety  of  contexts,  as  well  as  the  duality  the¬ 
ory  of  ordinary  linear  programming. 
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lu 


=  1 


2vl  +  2v2 


(3) 


- x  =  6/7 


3vl  +  2V2 


lu  -  1 


4vl  +  lv2 


and  so,  as  a  byproduct  of  our  calculations,  we  may  observe  that  both  DMU^  and  DMU^  are  charac- 

*  * 

terized  as  efficient.  I.e.,  they  have  values  of  unity  for  their  hQ.  The  hQ  =  6/7  for  DMl^  (which 

will  also  appear  as  the  optimal  value  of  the  functional)  means  that  it  is  inefficient.  This 
* 

value  of  h  =  6/7  is  also  accorded  the  operational  significance  of  meaning  that  under  effi¬ 
cient  production  only  6/7  of  the  amount  of  each  input  utilized  by  DMl^  should  have  been  re¬ 
quired.  Alternatively,  the  rec iprocal  value  could  have  been  employed  to  mean  that  7/6 
units  of  output  could  have  been  secured  in  place  of  the  one  unit  actually  realized 
from  the  input  amounts  utlized  by  DMl^- 

Reference  to  Figure  1,  below,  will  provide  some  easy  extensions 
that  will  also  further  help  us  to  interpret  and  evaluate  the  results 
we  have  now  achieved.  Here  DMU^,  DMl^  and  DMU^  are  accorded  interpreta¬ 
tions  as  points  in  a  two-dimensional  space  of  the  inputs  associated  with 
and  X£,  respectively.  The  solid  line  connecting  (=  DMU^)  and 
(=  DMU3)  represents  the  so-called  "unit  isoquant"  along  which  efficient 
production  should  occur  when  only  one  unit  of  output  is  produced. 

Evidently  (=  DMl^)  does  not  lie  on  this  isoquant.  The 


point 

obtained  from 

6  r,  6  (  3 

7  18/7'. 

t 

(4.1) 

11  7  '  2/ 

=  1  12/7)  ‘ 

P2 

does. 

however,  lie 

on  this  line.  This  is 

interpreted 

to  mean 

that 

under  efficient 

production 

DMU2  should 

have  been  . 

able 

to  cut  the  amounts 

of  each 

input 

to  6/7  of 

its 
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observed  value  in  achieving  one  unit  of  output.  Alternatively,  we  can 
use  the  v  values  of  (2.2)  in  order  to  substitute  in  the  denominator  of  the 
second  expression  —  i.e. ,  the  expression  for  DMl^  —  in  (3)  to  obtain 
(4.2)  3vJ  +  2v“  =  7/6  . 

In  this  form  we  are  saying  that  DMl^  should  have  obtained  7/6  units  of 
output  (or  output  rate)  from  the  amounts  of  the  inputs  utilized. 

To  appreciate  the  sense  of  what  is  being  said,  we  may  observe 

*  * 

that  these  v^  and  v^  values  represent  the  "marginal  productivities"  asso¬ 
ciated  with  the  inputs  to  which  they  are  assigned  in  this  single  input  caser^ 
These  are  efficient  marginal  productivities,  however,  obtained  by  reference 
to  the  data  of  (=  DilU^)  and  (=  DMU^).— ^  In  short  these  v^  and  v^ 
values  are  the  efficient  marginal  productivities. 

The  production  surface  from  which  the  isoquant  in  Figure  1  is 
derived  is  piecewise  linear.  It  follows  that  for  each  such  segment  we 
have  the  result  that  marginal  and  average  productivities  are  equal.  Thus 
the  v^  and  v^  values  obtained  from  (1)  provide  an  estimate  of  the  total  product 
that  should  have  been  obtained  if  the  efficient  productivities  were  applied 
to  the  input  data  of  DMU^  as  in  (4.2). 


^Proofs  of  this  proposition  are  provided  in  [9]  and  [26]. 

2 

Observe,  in  particular,  that  these  values  are  not  the  marginal  produc¬ 
tivities  observed  for  DMIL.  For  further  discussion  and  accompanying 
proofs,  see  [  i  ]  and  [:.]. 
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Turning  again  to  Figure  1  we  may  imagine  that  P,.  and  repre¬ 
sent  data  for  two  additional  DMU's.  That  is,  continuing  with  the  same  inputs 
and  same  outputs  as  the  Kills  in  Table  2  we  imagine  that  the  data  set  is  ex¬ 
panded  to  include  DMU^  and  DMU^  while  retaining  the  data  for  DMU^,  DMl^  and 


DMU, 


Singling  out  DMU^  with  values  for  its  input  coordinates  as  represented  in 
Figure  1,  we  use  all  of  the  data  shown  in  this  Figure  to  rate  the  unit's  efficiency  via 

lu 


max.  h  =  _  .  0 

o  2v,  +  3v„ 


subject  to: 


■ 


(5) 


1  > 


lu 


2vl  +  2v2 


1  > 


lu 


-  3v^  +  2v2 


1  -  lu 
1  -  4v”  +  iv2 


1  > 


lu 


-  2Vj  +  3v2 


i  >  _ lu 

-  lv-^  +  4v2 


u,  vp  v2  >  0 


This,  we  may  observe,  simply  extends  the  formulation  in  (1)  by  (a)  adjoin¬ 
ing  the  two  new  constraints  associated  with  the  data  for  DMU^  and  DMUj  and 
(b)  replacing  the  previous  functional  to  be  maximized  for  rating  DMU2  with  the 
new  one  for  rating  DMU^. 


I 
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(6.1) 


Proceeding  as  before  to  obtain  a  solution  to  (5)  we  have: 

h*  =  6/7 
o 


along  with  the  optimal  weights 


(6.2) 


u*  -  1,  v*  -  1/3,  v*  -  1/6. 


Substituting  in  (5)  we  see  that  all  constraints  are  satisfied  -  viz. , 


*^  =  1 

2Vj  ♦  2»2 


3»1  ♦  2v2 


=  3/4 


4vl  +  lv2 


*  =  2/3 


=  6/7 

2v^  +  3v2 


=  1  . 


lvl  +  ^v2 

We  now  make  the  Important  observation  that  the  efficient  reference  set 

consists  of  (■  DMU1)  and  (■  DMU^)  which  are  the  only  DMUs  achieving  a  value 

of  unity  as  a  result  of  the  optimization  in  (5).  Comparing  the  v  values  in  (6.2) 

* 

and  (2.2)  we  see  that  these  have  altered.  Thus,  although  DMU^  has  a  value  hQ»6/7 

the  same  as  DMU2  in  (2.1),  we  may  also  observe  that  DMU2  has  a  value  of  3/4 

* 

in  the  solution  provided  by  (7).  The  former  value,  e.g.,  ho*6/7,  is  the  correct 
rating,  however,  for  DMU. and  is  obtained  from  its  reference  set  consisting  of 
and  P^«  The  points  P^  and  P^  in  (7)  are  the  correct  reference  set  for  DMU^  but  not 
DMU2 .  Thus,  we  must  be  sure  to  obtain  the  efficient  marginal  productivities  from 
the  currect  reference  set.—  Simply  averaging  or  regressing  across  the  entire  effi¬ 
cient  set  will  not  do.' 

"The  way  in  which  this  extends  the  Marshallian  "representative  firm"  concept  is  dis¬ 
cussed  in  [9]. 


i 
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We  may  now  summarize  the  results  we  have  achieved  by  formally 

writing 

(8.1)  {(x1(x2)  :  ^  +  |  x2  =  1;  2  <  x1  <  4;  1  <  x2  <  2}  , 

for  the  efficient  isoquant  segment  that  extends  between  P^  and  P2-  This, 
in  turn,  differs  from 

(8.2)  |(x1(x2)  :  |  x1  +  i  x2  =  1;  1  <  Xj^  <  2;  2  <  x2  <  4}  , 

for  the  segment  that  extends  between  P^  and  P  in  the  "piecewise  linear" 
fashion  that  is  exhibited  in  Figure  1. 

Observe  that  we  have  now  also  provided  a  new  way  for  effecting 
estimates  of  economic  relations  from  empirical  data.  These  differ  from 
the  ordinary  statistical  approaches  such  as  'regression' and  “simultaneous 
equation"  estimation  in  various  ways.  The  latter,  for  instance, 

focus  on  averages  and  other  measures  that  are  in  the  interior  of  the 
observations  from  which  estimates  are  to  be  effected.  Our  measures  are 
directed  to  extremals,  or  extremal  relations,  that  lie  rather  on  the 
boundaries  rather  than  in  the  interior  of  these  same  observations. 

It  must  be  expected  that  uses  of  these  new  methods  of  estima¬ 
tion  may  also  differ  in  important  ways  from  usages  that  are  associated 
with  the  more  customary  techniques  of  statistical  regression.  Note,  for 
instance,  that  the  slope  conditions  in  (8.1)  and  (8.2)  assume  that  the 
efficient  boundary  is,  or  can  be  made,  applicable  to  DMUs  which  lie 
within  the  regions  defined  by  the  inequalities  in  each  set.  This  is  to 
say  that  two  possible  sources  of  economizing  are  present.  One  source 

consists  of  economizing  on  inputs  or  expanding  the  outputs  by  bringing 

*  1/ 

each  DMU  onto  the  efficient  surface  in  accordance  with  its  h  valueT 

o 

^Because  it  permits  an  expansion  of  output  without  requiring  an  increase  in  any 
input,  this  is  technically  identified  as  '\*aste" —  in  order  to  distinguish  it  from 
other  types  of  economizing  that  may  also  be  possible. 
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The  other  source  consists  of  effecting  substitutions  between  various  inputs  in  ac¬ 
cordance  with  prices  or  other  conditions  for  the  applicable  ranges,  as  exhibited  in 
each  of  (8.1)  and  (8.2)  af  ter  the  assoc  iated  DMU  has  been  brought  into  efficient  operations . 

We  have  elsewhere  distinguished  this  new  method  for  estimating 
extremal  relations  by  referring  to  it  as  a  "control  prediction."^  That  is, 
in  contrast  to  the  "pure  predictions!'  which  we  associate  with  ordinary 

social  science  usages  of  statistical  regressions,  etc.,  such  control  predictions 
assume  that  mechanisms  exist  for  bringing  about  the  wanted  states  of  effi¬ 
cient  production. 

Here  we  would  like  to  associate  these  kinds  of  "control  prediction" 
results  with  procedures  such  as  the  "analytic  Leviews,"  etc.,  which  are  often  used 
to  guide  the  planning  of  audit  strategies  in  both  financial  and  comprehensive-type 
audits.  Thus  we  might,  for  instance,  assemble  data  on  a  set  of  public  schools  — 
along  lines  such  as  we  shall  soon  be  discussing  —  in  order  to  direct  audit  efforts 
to  more  detailed  attention  at  specified  efficient  or  inefficient 
sites. 

Such  "control  predictions"  are,  of  course,  not  entirely  new  to 
accountants.  They  may  be  found,  for  instance,  in  areas  like  engineered 
standard  costs.  Indeed  the  analogy  with  "efficiency  standards"  may  be  con¬ 
sidered  to  be  a  close  one  when  the  latter  is  associated  with  predicted  out¬ 
come'  that  .should  occur  when  prescribed  (efficient)  procedures  are  followed. 

Just  as  in  the  case  of  our  measure,  too,  an  efficiency  variance  does  not  include 
price  variations  explicitly  although,  of  course,  relevant  substitution  of  mater¬ 
ials  and  procedural  alterations  may  be  effected  when  prices  warrant  this. 

lSee  [  6  ] .  . 
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Farther,  the  appearance  of  a  black  or  red  efficiency  variance  serves  as  an 
attention  directing  device  for  further  inquiry —  perhaps  by  an  auditor — rather 
than  by  automatically  providing  the  specific  correctives  that  may  be  required. 

Finally,  we  may  observe  that,  as  is  also  true  for  standard  costs, 
our  efficiency  measure  is  intended  to  supply  new  options  that  were  previously 
not  available  for  policy  guidance  and  it  is  also  intended  to  open  still  further  possi¬ 
bilities  such  as  the  distinction  between  "program  efficiency"  and 
"managerial  efficiency"  that  we  shall  now  begin  to  explore. 


4.  DATA  ENVELOPMENT  ANALYSIS 


Figure  2,  below,  will  help  us  to  introduce  our  next  topic.  We  may 
think  of  this  as  a  portrayal  in  which  amounts  x  of  a  single  input  have 
resulted  in  amounts  y  of  a  single  output  (all  else  held  constant)  for  a 
collection  of  DMUs  with  the  resulting  observations  portrayed  as  in  Figure  2. 
There  is  a  further  distinction  in  that  these  DMUs  operate  under  two  differ¬ 
ent  technologies,  or  programs, so  that  the  efficient  frontiers  for  one  set 
are  at  B  while  the  efficient  frontiers  of  the  other  set  are  at  A.  Evidently 
even  efficient  DMUs  in  B  cannot  achieve  the  levels  of  those  in  A  and  even 
some  of  the  less  efficient  DMUs  operating  under  A  exceed  what  is  attainable 
under  B. 

Customary  approaches  as  in  a  use  of  statistical  regressions  might  have  concealed 
some  of  the  alternatives  for  choice  that  are  present  because  the  observations 
generally  contain  both  different  technological  possibilities  and  different 
degrees  of  efficiency  in  their  utilization.  This  is  the  situation  we  would 
now  like  to  confront  by  extending  our  previous  analysis  in  order  to  bring  it 
to  bear  in  ways  that  might  help  us  distinguish  between  what  we  shall  refer 
to  as  "managerial  efficiency"  and  "program  efficiency."—^  The  latter  may  be 
thought  of  in  terms  of  the  kinds  of  frontiers  depicted  in  Figure  2  while 
the  former  may  contain  inefficiencies  resulting  from  managerial  decisions 
that  fail  to  utilize  these  opportunities  to  the  full. 

"Data  Envelopment  Analysis"  is  the  name  that  we  shall  use  for  the 
approach  we  will  be  suggesting  because  (a)  it  first  tries  to  locate  the  boun¬ 
daries  that  envelop  the  observations  as  in  A  and  B  of  Figure  2  and  then  (b) 


LThe  re  may  be  valid  reasons  for  failure  to  utilize  efficiency  possibilities 
to  the  full,  as  R.  Lehto  has  observed  to  us,  but  even  then  the  approach  we 
are  suggesting  can  provide  measures  of  the  "cost"  of  the  actual  vs.  potential 
efficiencies  that  are  then  foregone. 
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it  brings  the  observations  all  up  to  the  envelope  that  is  pertinent  in  each 
case.  The  degree  of  "management  inefficiency,"  if  any,  is  first  measured 
relative  to  the  pertinent  A  envelope  or  B  envelope^in  the  manner  indicated 
in  the  preceding  section  for  each  observation.  The  remaining  efficiency 
differences  are  then  imputed  to  the  respective  programs  (or  technologies) 
so  that  in  the  situation  of  Figure  2,  say,  the  program  associated  with  A 
would  be  characterized  as  more  efficient  than  the  program  associated  with  B. 

In  the  preceding  section  we  provided  a  measure  of  efficiency  that 
had  operational  significance  when  the  managers  were  all  operating  under  a 
common  program.  It  is  evidently  also  desirable  to  try  to  do  this  in  our 
assessment  of  program  efficiencies  as  well.  That  is,  we  would  like  to  have 
a  means  of  taking  account  of  the  efficiency  gains  (or  losses)  that  can  be 
secured  by  moving  DMUs  from  one  program  to  the  other. 


"We  refer  to  these  more  generally  as  "a-envelopes"  in  order  to  allow  for  com¬ 
parisons  between  a=l,  2,  n  different  technologies  or  programs.  See  [8]. 


Plusos(+)  *  ions  l\u  UMl's  in  A 

Dots  (  •)  "  Ul- m-  i  v.it  Ions  t  or  in  H 


To  clarify  what  we  are  about,  we  may  again  refer  to  our  earlier 
contra. ,t  between  "control  predictions"  and  "pure  predictions."  A  variety 
of  possibilities  then  confront  us.  For  instance,  we  might  fit  two  separate 
statistical  regressions  of  the  ordinary  least  squares  variety  to  the  obser¬ 
vations  shown  in  Figure  2.  This  could  lead  us  to  conclude  that  the  B 
program  was  better  than  A  after  which  we  might  employ  the  resulting  B 
regression  to  estimate  how  the  DMUs  previously  operating  under  A  would 
perform  if  the  latter  were  eliminated  in  favor  of  B. 

However,  if  new  "controls"  could  be  instituted  to  alter  past 
patterns  of  behavior  then  a  different  prediction  (of  the  "control"  variety) 
might  better  be  employed.  In  particular,  if  these  controls  could  ensure 
that  all  DMUs  would  operate  at  the  program  boundaries  then  A  would  be 
preferred  to  B.1 

DEA,  Data  Envelopment  Analysis,  the  approach  we  shall  now  begin 
to  examine,  is  intended  for  use  in  the  latter  situations  since,  without  such 
controls,  the  resulting  "efficient  behavior"  predictions  will  generally  be 
invalid.  We  may,  in  fact,  think  of  DEA  as  providing  guidance  for  program 
audits  which  will  help  to  locate  sources  of  inefficiency  along  lines  such 
as  we  have  indicated  in  the  preceding  sections  of  this  paper  -  e.g.,  direct¬ 
ing  such  audit  examinations  to  particular  DMUs  brought  into  view  by  a  DEA 
analysis.  A  separate  decision  may  then  be  made  concerning  whether  the 
projected  efficiencies  can  be  realized  and  perhaps  thereby  justify  a  choice 
of  A  rather  than  B. 

Still  other  possibilities  would  allow  for  situations  in  which  only  some  of 
the  DMUs  might  be  brought  up  to  their  boundaries  while  others  are  only 
partially  improved,  etc.  The  point  is,  in  any  case,  that  our  DEA  approach 
opens  new  alternatives  for  consideration  besides  those  available  only  from  the 
"pure  predictions"  approach. 
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5.  THE  PROGRAM  FOLLOW-THROUGH  EXPERIMENT 

In  order  to  illustrate  what  is  involved  we  turn  to  data  from 
"Program  Follow-Through,"  a  large-scale  experiment  in  U.S.  public  school 
education  designed  to  help  disadvantaged  children.  This  program  was  in¬ 
tended  to  further  the  objectives  of  a  pre-school  program  known  as  Project 
Head  Start,  by  carrying  on  specially  designed  programs  for  disadvantaged 
children  from  kindergarten  through  grade  3.  This  was  done  at  the  70  selected 
"school  sites"  in  various  parts  of  the  U.S.  which  are  listed  in  Table  A-l. 
(See  first  two  columns  of  Table  A-l  for  the  numbering  that  will  hereafter  be 
identified  with  these  sites.) 

An  assessment  of  Program  Follow-Through  was  sought  in  what  was 
intended  as  a  designed  experiment^  by  selecting  a  matched  pair  of  Program 
Follow-Through  and  Non-Follow-Through  sites  from  which  observations  were  to 
be  secured.  These  are  identified  as  PFT  and  NFT,  respectively,  in  Columns  1 
and  2  of  Table  A-l.  Although  we  need  to  observe  that  the  experiment  was  not 
carried  out  successfully  in  all  respects,  it  nevertheless  produced  a  wealth 
of  data  from  which  we  effect  a  selection  as  follows. 

We  restrict  ourselves  only  to  educational  aspects  of  the 
program  while  ignoring  other  parts  of  its  activities  such  as  supplying 
medical  and  dental  services,  nutritional  programs,  etc.  We  also  omit  any 
explicit  consideration  of  costs  on  the  supposition  that  a  cost/benefit 
analysis  only  becomes  pertinent  if  statistically  significant  effects 
are  obtained  from  program  elements  such  as  we  are  considering. 


Or  at  least  as  close  to  such  an  experiment  as  one  is  likely  to  achieve  in 
public  school  situations. 
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Table  A-l 
(continued) 


*NE  ■  North  Eastern  United  States 
S  ■  Southern  United  States 
NC  “  North  Central  United  States 
W  ■  Western  United  States 


2 

Large  City 
Medium  City 
Small  City 
Rural  Area 


200,000  or  more 
50,000  to  199,999 
10,000  to  49 ,999 
Less  than  10,000 


All  Data  Envelopment  Analysis  study  information  refers  to  the  Cohort  II-K 
student  population.  II-K  indicates  that  this  group  of  students  began  their 
Program  Follow  Through  experience  in  kindergarten.  (This  was  also  the  only 
one  of  three  Cohorts  which  had  completed  all  of  the  grades  from  kindergarten 
through  third  grade  at  the  time  of  our  study  for  which  site  level  information 
was  available,)'  However,  due  to  incomplete  statistics  along  some  DEA 
variable  dimensions,  some  of  the  Cohort  II-K  PFT  sites  were  not  included 
in  the  DEA  study.  Specifically,  Bank  Street  Model:  Rochester, NJ  site; 

EDC  Model:  Chicago, IL  site;  and  SEDL  Model:  St.  Martin  Parish, LA  site 
were  excluded  from  the  DEA  study  student  population.  The  actual  Cohort  II-K 
PFT  population  was  3,367  of  which,  as  noted  above,  a  set  of  3,210  students 
were  used  in  the  DEA  study.  This  exclusion  of  sites  also  extended  to  the 
NFT  groups  which  were  similarly  reduced  to  1,202  students. 


Two  sets  of  NFT  students  groups  were  created  in  the  original  Program 
Follow  Through  study.  One  group  was  a  local  student  set,  usually  in  the 
same  school  system  as  the  subject  PFT  site.  The  secaffd  group,  and  the 
one  selected  for  the  DEA  study,  was  a  "best  matched"  group,  which  may  or 
may  not  have  been  located  in  the  same  school  system  or  even  the  same 
geographical  region.  The  NFT  group  which  nx>st  nearly  matched  the  PFT  students 
of  a  given  site  along  a  number  of  demographic  and  initial  performance 
dimensions  was  considered  the  "best  match"  for  the  latter.  For  several 
PFT  sites  the  same  "best  matched"  NFT  group  was  used.  The  much  smaller 
NFT  student  population  total  of  1,202  as  compared  to  the  PFT  student 
total  of  3,210  resulted.  See  also  preceding  footnote. 
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We  shall  focus  on  only  one  of  several  cohorts  from  the  subjects 
comprehended  in  the  study.  In  addition,  we  shall  utilize  only  the  terminal 
(grade  3)  results  for  this  cohort  to  avoid  the  additional  complications 
needed  to  deal  with  dynamic  or  transient  behavior  en  route  to  this  terminus.^ 
From  a  set  of  11  output  measures  we  select  only  the  following  3  as  suffi¬ 
ciently  indicative  for  our  purpose: 

y^:  Total  Reading  Score  as  measured  by  the  Metropolitan 
Achievement  Test. 

(9)  y^:  Total  Mathematics  Score  as  measured  by  the  Metro¬ 

politan  Achievement  Test. 

y^:  Coopersmith  Self-Esteem  Inventory,  intended  as  a 
measure  of  self-esteem. 

This  y^  measure,  we  may  note,  is  directed  to  non-cognitive  growth 
(or  affective  behavior)  in  a  dimension  that  is  deemed  pertinent  to  the 
objectives  of  this  program.  Together  with  y^  and  y 2,  this  y^  variable 
provides  a  good  indication  of  what  is  involved  in  assessing  such  programs. 
Note,  in  particular,  that  no  easily  available  scheme  for  weighting  the 
relative  importance  of  these  outputs  is  at  hand.  Nevertheless,  some 
"overall"  measure  of  program  efficiency  is  wanted  in  order  to  enable  us 
to  evaluate  PFT  vs.  its  NFT  alternative  and  this  is  to  be  achieved 
from  data  such  as  are  exhibited  in  Tables  A-2  and  A-3. 


More  detailed  developments  of  the  approach  we  are  examining  may  be 
found  in  (  8  )  and  [26] . 


2r 

Table  A-? 


Unadjusted  PFT  Output  Observations 


SUt  t  Total  Reading 

Score*, PHS* 


Total  Math 
Score* ,  PUS* 


Total  Coopersaith 
Scoraa.PHS* 


1 

54.53 

58.98 

38.16 

2 

24.69 

33.89 

26.02 

3 

36.41 

40.62 

28.51 

A 

14.94 

17.58 

16.19 

3 

7.81 

6.94 

5.37 

S 

12.59 

16.83 

12.84 

7 

17.06 

16.99 

17.82 

a 

20.29 

30.64 

33.16 

9 

26.13 

29.80 

26.29 

10 

46.42 

51.59 

35.20 

11 

39.80 

37.73 

30.29 

12 

37.84 

47.85 

23.35 

13 

26.48 

31.36 

26.56 

14 

10.31 

10.86 

7.47 

15 

14.39 

18.30 

14.33 

16 

32.94 

36.03  . 

38.19 

17 

17.25 

20.80 

12.07 

IS 

27.53 

38.19 

20.44 

19 

41.12 

43.80 

36.54 

20 

29.43 

42.63 

23.34 

21 

37.46 

51.02 

27.44 

22 

19.40 

25.11 

16.52 

23 

39.88 

47.72 

38.97 

24 

23.72 

30.81 

16.54 

25 

24.88 

25.27 

22.43 

26 

31.62 

40.78 

31.16 

27 

31.31 

38.32 

25.03 

28 

21.00 

21.30 

16.30 

29 

6.51 

7.02 

6.16 

30 

11.64 

15.26 

15.68 

31 

12.58 

15.90 

14.42 

32 

4.59 

6.16 

4.99 

33 

43.76 

46.64 

39.10 

34 

32.38 

38.55 

31.05 

35 

34.64 

43.46 

39.22 

36 

11.52 

15.14 

13.91 

37 

15.96 

19.21 

15.30 

38 

9.91 

12.30 

7.22 

39 

30.44 

33.53 

29.80 

40 

22.63 

25.24 

37.35 

41 

24.41 

27.16 

25.30 

42 

23.11 

22.67 

27.56 

43 

21.82 

31.45 

27.54 

44 

63.92 

79.67  . 

63.  U 

45 

9.47 

11.92 

8.65 

46 

33.94 

39.18 

34.61 

47 

29.42 

35.10 

28.42 

48 

7.70 

11.02 

9.02 

4* 

12.17 

16.03 

15.82 

*  P11S  •  Per  Hundred  Student* 
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Table  A- 3 


% 


Unadjusted  PFT  Input  Observations 


Education  Level  Occupation 

of  Mo Jter.PHS*  Index, PUS* 


Parental  Visit 
Index, PUS* 


Cowseling  Muabar  of 

**to*>T*S  Teacher* 


*1. 


*2 


*«  *5 


1 

86.13 

2 

29.26 

3 

43.12 

4 

24.96 

5 

11.62 

6 

11.88 

7 

32.64 

8 

20.79 

9 

34.40 

10  . 

61.74 

11 

52.92 

12 

36.00 

13 

39.20 

14 

14.6 

15 

4.29 

16 

27.25 

17. 

22.63 

18 

28.00 

19 

53.36 

20 

25.42 

21 

31.37 

22 

16.34 

23 

44.28 

24 

19,74 

23 

24.40 

26 

41.40 

27 

27.20 

28 

23.92 

29 

10.62 

30 

12.48 

31 

19,32 

32 

6.30 

33 

46.62 

34 

38.95 

35  • 

61,60 

36 

31.08 

37 

19.35 

38 

11.20 

39 

34.40 

40 

35.55 

41 

30.53 

42 

25.44 

43 

26.66 

44 

39.79 

43 

8.32 

46 

59.78 

47 

39.22 

48 

3.24 

49 

7.14 

16.24 

10.24 
11.31 

0.14 

2.21 

4.97 

6.88 

12.97 

11.04 

14.10 
11.67 

10.13 
10.80 

2.88 

3.42 
14.17 

4.43 
7.61 

13.70 
9.03 

10.08 

5.84 

14.14 

6.43 
8.03 

11.70 
9.38 
7.12 
2.33 
6.14 
5.89 
1.93 

14.65 

12.82 

13.56 

6.26 

6.68 

3.08 

11.61 

6.48 

9.30 

7.10 
11.43 
22.49 

3.64 

13.52 

10.06 

3.18 

3.29 


48.21 

41.96 

38.19 

24.61 

6.85 

18.73 
28.10 
34.65 
38.16 
49.09 
39.48 
37.80 
41.04 

9.64 

21.45 

36.46 
15.40 

28.73 
33.04 

29.69 
39.34 
20.89 

36.70 
24.20 
33.42 
44.01 

37.80 
25.58 
10.10 
23.13 
24.01 

7.11 

65.71 
47.02 
53.98 
22.18 
22.61 

9.90 

41.79 
21.69 
35.50 

26.81 
41.36 
64.77 
12.92 

48.80 
37.00 
13.12 
23.10 


49.69 

40.63 
35.03 

23.13 
6.37 

18.04 

23.43 

52.07 

42.40 

42.92 

39.64 
39.32 
41.12 

11.14 
17.27 

53.26 
13.00 
27.04 
49.83 

31.74 
40.37 
22.10 

32.27 
25.66 

31.29 
46.35 
31.53 
29.01 

9.09 

22.46 

24.74 
7.68 

57.49 

48.92 

50.29 
21.96 
23.31 
10.06 
41.79 

21.69 
35.14 
26.23 
44.63 

76.12 

13.13 

49.69 
38.33 
12.71 
19.06 


11 


11 

2 

15 


♦ 


•PUS  •  Per  Uundred  Student* 
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The  latter  table,  i.e.,  A-3,  contains  the  input  data  for  those  same 
PFT  sites  in  terms  of  the  following  variables: 


x^:  Education  level  of  mother  as  measured  in  terms  of 
percentage  of  high  school  graduates  among  female 
parents . 

x„ :  Highest  occupation  of  a  family  member  according  to 
a  pre-arranged  rating  scale. 

x^:  Parental  visit  index  representing  the  number  of 
visits  to  the  school  site. 

x^:  Parent  counseling  index  calculated  from  data  on  time 
spent  with  child  on  school-related  topics  such  as 
reading  together,  etc. 

Xji  Number  of  teachers  at  a  given  site. 


These  variables,  like  the  output  variables,  are  all  measured  in 

terms  of  scores  per  hundred  students  with  the  exception  of  x^,  the  number 

of  teachers,  which  is  measured  without  further  adjustment  as  given  at  each 
1 

site. 


The  value  of  hQ  as  a  measure  of  relative  efficiency  is  not  affected  by 
such  scale  choices  provided  different  scales  are  not  used  at  different  sites  for 
any  particular  input  or  output.  See  [6]  and  [26]. 
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The  above  variables  represent  a  selection  from  25  input  variables 
which  together  with  the  11  output  variables  define  the  effectiveness  sought 
for  the  educational  components  of  Program  Follow-Through.  Most  of  the  inputs 
are  given  so,  in  fact,  the  measures  of  eefectiveness  are  largely  in  terms  of  the 
output  scores  that  should  be  attainable  with  the  input  mixes  available  at 
each  of  the  sites.  This,  in  turn,  is  dependent  on  the  efficiency  with 
which  these  resources  are  utilized  —  which  we  shall  now  measure  along  the 

lines  of  the  more  restricted  data  sets  that  we  have  already  indicated. 

Turning  to  the  data  in  Tables  A-2  and  A-3  we  might  proceed  to 
rate  the  efficiency  of  site  1,  say,  relative  to  the  data  for  all  49  PFT 
sites  by  extending  our  previous  analysis.  This  would  yield  the  following 
expression: 

54.53^  +  58.98u2  +  38.16u3 

86.13v3  +  16.24v2  +  48.21v3  +  49.69v^  +  9v^ 

54.53u^  +  58.98u2  +  38.16u3 

86.13v  +  16.24v.  +  48.21v~  +  49.69v.  +  9vc 

1  2  3  4  5 

24.69u1  +  33.89u2  +  26.02u3 

29.26v..  +  10.24v„  +  41.96v_  +  40.65v,  +  5vc 

1  2  3  4  5 

1/ 


max .  h  = 
o 

subject  to: 

1  > 

(11.1)  1  > 


12 . 17u1  +  16.03u2  +  15.82u3 

1  -  7 . 14v.  +  5 . 29v„  +  23 . 10v_  +  19.06v,  +  8vc 

i  2  3  4  5 

V  u2’  U3’  vl»  v2 ’  v3 *  v4»  v5  >  0 
Now  we  may  observe  that  we  are  confronted  with  a  situation 
involving  multiple  outputs  and  hence  the  simple  concept  of  a  production 
function  with  its  related  efficient  marginal  productivities  is  no  longer 

^The  ellipsis  " . "  means  that  the  46  omitted  expressions  are  to  be  formulated 

in  the  manner  indicated  by  the  expressions  that  are  written  explicitly. 


r 


\ 


F  J 
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directly  available.  We  may  instead  think  of  the  numerator  as  defining 

a  new  output  and  the  denominator  as  defining  a  new  input  which  are  both 

scalar  (single  valued)  numbers,  that  we  shall  refer  to  as  the  "virtual 

output"  and  the  virtual  input,"  respectively.  In  any  case  the  value  of 

h  is  limited  to  lie  between  0  and  1,  as  before, 
o 

In  a  similar  way  we  refer  to  Tables  A-4  and  A-5  to  achieve 
efficiency  ratings  for  the  NFT  sites.  Thus,  for  instance,  for  site 
number  70  we  would  utilize  the  following  to  obtain  its  hQ  efficiency 
rating:—^ 

13.69u1  +  14.19u2  +  12.99u3 

maX-  ho  =  10 . 44v.  +  4.82v_  +  17.13v,"+  18.21v,  +  9vc 
1  2  3  4  5 

subject  to: 


39.07^  +  42.71u2  +  27.67u3 
-  68.16Vl  +  12_.28v2  +  33.58v3  +  34.64v4  +  15v3 


(11.2) 


9.96Ul  +  14.34u2  9.33u3 

1  -  11.'88v7  +  3.59V,  +  13.41v,  +  13.82v,  +  8v, 
1  2  3  4  5 


13.69ux  +  14.19u2  +  12.99u3 

1  -  10 .44v.  +  4.82v-  +  17.13V,  +  18.21v.  +  9v. 

1  2  3  4  5 

u^,  u2,  u3,  v^ ,  v2,  v3,  v^,  v^  >  0 


Note  that  the  constraints  remain  the  same  in  all  cases  and  only  the  expression 
in  the  function  being  maximized  changes  .  This  feature  can  be  utilized 
moreover,  in  the  computing  routines  by  allowing  for  an  alteration  between  the 
"simplex"  and  "dual  methods"  as  described  in  [26]. 
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Table  A-  4 


Unadjusted  NFT  Output  Observations 


81ta  # 


50 

51 

52 

53 

54 

ss 

56 

57 
SO 

59 

60 
61 
62 
6] 

66 

63 

66 

67 

66 

69 

70 


Total  Reading 
Score*  ,PHS* 

*1 

39.07 

9.96 
65.37 
18.23 
39.63 
26.20 
13.53 
26.39 
21.67 
‘  120.17 
15.15 
6.92 
9.35 
13.03 
18.63 
12.28 
16.81 
26.36 
22.85 
8.17 
13.69 


Total  Math 
Score*  tPH$* 


Total  Cooparaalth 
tear**  .PUS* 


62.71 

27.67 

16.36 

» 

9.33 

51.38 

31.61 

22.05 

17.56 

66.61 

35.89 

28.21 

18.76 

17.09 

15.61 

27.65 

• 

20.79 

26.22 

13.66 

166.67 

88.59 

18.06 

13.58 

7.10 

6.35 

9.85 

• 

7.70 

13,60 

10.29 

26.48 

23.13 

13.01 

9.89 

19.72 

18.70 

28.22 

26.66 

26.21 

28.16 

8.70 

5.12 

16.19 

12.99 

Hundred  Student* 
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Tabic  A~5 


Unadjusted  NFT  Input  Observations 


ttM  I 

education  Level 
of  Mother, rttS* 

Occupation 
Index,  PUS* 

Parental  Vlalt 
Index, PUS* 

Counseling 

Index.PHS* 

Muafcer  of 
Teachers 

*1 

*2 

*3 

*4 

*5 

50 

68.16 

12.28 

33.58 

34.64 

15 

Si 

11.88 

3.59 

13.41 

13.82 

8 

SI 

5S.30 

11.53 

36.73 

35.78 

6  • 

S3 

16.20 

7.02 

26.94 

26.30 

9 

54 

82.45 

15.52 

45.00 

44.23 

13 

55 

15.81 

6.93 

23.91 

23.61 

7 

S« 

4.65 

5.50 

20.91 

23.39 

5 

57 

41.25 

8.41 

26.23 

25.24 

10 

58 

10.44 

5.22 

17.10 

18.93 

3 

53 

139.65 

35.03 

119.56 

130.83 

22 

60 

16.28 

4.81 

18.20 

18.98 

5 

<1 

12.06 

2.59 

8.74 

8.17 

5 

u 

4.20 

2.64 

9.89 

11.25 

2 

63 

19.44 

3.83 

12.87 

13.23 

5 

64 

28.38 

8.91 

30.95 

33.33 

8 

65 

13.50 

3.61 

13.60 

12.39 

4 

66 

23.32 

7.10 

24.96 

28.36 

22 

67 

27.60 

9.38 

32.29 

34.01 

20 

66 

11.70 

10.53 

37.67 

43.60 

8 

69 

4.68 

1.85 

6.22 

3.46 

3 

70 

10.44 

4.82 

17.13 

18.21 

» 

•rus 

•  fer  Hundred  Student* 

• 
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Repeated  applications  of  the  above  procedures  yields  the  results 

listed  in  Table  A-6.  Note  that  the  efficiency  ratings  are  with  respect  to 

•k 

each  of  the  programs,  PFT  and  NFT,  respectively.  Thus  the  hQ  =  1  for  site  1 

* 

is  obtained  relative  to  PFT  sites  only,  while  the  hQ  =  0.94  for  site  70  is 
obtained  relative  to  NFT  sites  only. 
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Tabic  A- 6 


PFT  and  NFT  Program  Specific  a-Envelope  Efficiency  Values 


*1 

PFT  h0 

Site  #  Efficiency  Value 


*2 

nft  h0 

Site  #  Efficiency 


1* 

1.00 

2 

0.90 

3 

0.98 

4 

0.90 

5* 

1.00 

6 

0.90 

7 

•  0.89 

8 

0.91 

9 

0.87 

10* 

1.00 

11 

0.98 

12 

0.97 

13 

0.86 

14 

0.98 

IS* 

1.00 

16 

0.95 

17* 

1.00 

18* 

1.00 

19 

0.95 

20* 

1.00 

21* 

1.00 

22* 

1.00 

23 

0.96 

24* 

1.00 

25 

0.97 

26 

0.93 

27* 

1.00 

28 

0.94 

29 

0.84 

30 

0.90 

31 

0.83 

32 

0.90 

33 

0.94 

34 

0.85 

35* 

1.00 

36 

0.60 

37 

0.94 

38 

0.94 

39 

0.91 

40* 

1.00 

41 

0.94 

42 

0.94 

43 

0.87 

44* 

1.00 

45 

0.89 

46 

0.90 

47* 

1.00 

48* 

1.00 

49* 

1.00 

50 

0.95 

51 

0.92 

52* 

1.00 

53 

0.87 

54* 

.  1.00 

55* 

1.00 

56* 

1.00 

57 

0.92 

58* 

1.00 

59 

0.92 

60 

0.98 

61 

0.88 

62* 

1.00 

63 

0.96 

64 

0.91 

65 

0.97 

66 

0.92 

68* 

1.00 

69* 

1.00 

70 

0.94 

•  Denotes  a  site  with  an  efficiency  value  of  "l* 


Value 
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6.0  PROGRAM  AND  MANAGEMENT  EFFICIENCY  UNDER  PEA 
As  a  first  step  in  our  analysis  we  might  try  to  ascertain  whether  man¬ 
agerial  efficiencies  differ  under  PFT  and  NFT.  With  DEA  this  is  done,  we  may 
recall,  by  separately  obtaining  the  efficiency  ratings  for  the  DMUs  under  each 
of  PFT  and  NFT  by  reference  to  their  respective  envelopes,  and  this  is  the  way 
the  data  in  Table  A-6  were  obtained. 


For  convenience  of  reference  we  now  depict  the  data  of  Table  A-6  as  in 
Figure  3.  The  latter  can  be  interpreted  in  terms  of  frequency  diagrams  from 
which  a  statistical  comparison  between  PFT  and  NFT  might  be  effected  —  e.g., 
to  ascertain  whether  the  two  differ  sufficiently  so  that  we  might  reasonably 
reject  the  hypothesis  that  there  is  no  difference  in  the  managerial  effici¬ 
ency  in  these  two  distributions.  We  shall  not  undertake  such  an  analysis 
in  detail,  however,  since  this  would  divert  us  into  technical  statistical  con¬ 
siderations  and  away  from  our  main  objectives  Instead  we  call  attention  to 
the  ratios  associated  with  the  spikes  on  the  right  of  each  distribution  and  ob¬ 
serve  that  these  ratios  do  not  differ  significantly.  That  is,  there  is  no  sta¬ 
tistically  significant  difference  between  the  proportion  of  DMU's  that  achieve 
an  efficient  rating  in  PFT  and  NFT,  respectively. 


A  formulation  which  may  be  regarded  as  canonical  for  the  unusual  types  of  den¬ 
sity  functions  involved  in  these  distributions  may  be  found  in  the  Appendix  to 
[  6].  which  is  coauthored  with  E.  Rhodes. 


-  35  - 
Figure  3 

DISTRIBUTIONS  OF  EFFICIENCIES 


FOLLOW  THROUGH 
(PFT) 


30 


40 


50 


60 


70 


-  36 


This,  of  course,  does  not  end  the  analysis,  since  we  are  also 
interested  in  comparing  the  programs  per  se.  That  is  we  also  want  to 
effect  a  comparative  assessment  of  PFT  and  NFT  on  the  assumption  that  the 
DMUs  in  each  are  all  efficiently  operated. 

This  brings  into  play  the  next  part  of  our  DEA  approach  in  a  way  that  we  can 
illustrate  with  the  aid  of  Figure  4.  This  diagram,  which  is  purely  hypothetical  ,  may  be  inter¬ 
preted  in  a  manner  analogous  to  Figure  l,except  that  we  are  here  considering  PFT  and  NFT  iso¬ 
quants  in  the  same  diagram.  Referring  therefore  to  observations  such  as  A,  we  may  charac¬ 
terize  it  as  inefficient  relative  to  the  PFT  envelope  while  a  point  such  as  C  is  cha¬ 
racterized  as  inefficient  relative  to  the  NFT  envelope. 

The  latter,  i.e.,  the  PFT  and  NFT  envelopes  are  derived  in  the 
manner  we  have  already  indicated.  To  the  left  of  the  point  where  they  inter¬ 
sect  the  solid  line  associated  with  the  PFT  envelope  is  evidently 
less  efficient  than  the  solid  line  associated  with  NFT— ^  and  the 
opposite  situation  holds  thereafter.  In  such  a  situation,  one  may  conclude 

that  with  efficient  operations  the  NFT  program  may  best  be  used  under  one 

2 

set  of  conditions  and  PFT  under  another  set  of  conditions. 

Something  more  than  this  may  be  wanted,  however,  in  the  form 

of  an  "overall"  evaluation  of  PFT  and  NFT.  For  this  purpose  we  arrange 

an 'tnter-Program  Envelope"  which  we  abbreviate  to  "Inter-Envelope"  such  as  the  one 

portrayed  by  the  connected  series  of  broken  line  segments  that  are  also 

y 

shown  in  Figure  4.—  Observe  that  this  broken  line  isoquant  is  coincident 
with  NFT  in  some  sections  and  that  it  is  coincident  with  PFT  in  other 
sections.  In  no  case  is  it  ever  less  efficient  than  either.  In  other 

^Because  to  the  left  of  the  intersection  the  NFT  envelope  (=  isoquant) 
indicates  that  the  same  level  of  output  is  achieved  with  a  smaller 
amount  of  at  least  one  input. 

2 

For  further  discussion  and  references  see  [8  ]. 

3 

These  are  also  distinguished  as  "each"  and  "every"  evelopes  in  the  terminology  of 

[26] 
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words,  the  inter-envelope  is  intended  to  provide  a  standard  of  what  is 
best  attainable  in  the  sense  of  a  boundary  to  which  efficient  DMUs  would 
push  under  either  NFT  or  PFT  as  the  efficiency  situation  might  indicate. 

Of  course,  we  also  need  to  take  account  of  the  number  of  DMUs 
that  would  thereby  be  improved  as  well  as  the  amount  of  improvement  in 
each  case.  For  this  we  first  enforce  the  assumption  of  efficiency  by 
bringing  every  DMU  onto  its  program  boundary  and  then  assess  the  further 
gain  that  may  be  effected  by  allowing  the  thus  adjusted  observations  to 
move  from  the  boundaries  imposed  by  their  respective  programs  to  the 
(possibly)  still  further  efficier.cir  associated  with  the  inter-envelope. 


i 
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The  methods  for  effecting  these  adjustments  have  been  set  forth 

in  detail  in  references  such  as  [ 9  1  and  { 26] . ^  Hence  we  do  not  need  to 

do  more  than  illustrate  what  is  involvtd  by  reference  to  (12),  below. 

Here  we  are  supposing  that  site  number  1  is  to  be  rated  by  reference  to 

the  entire  set  of  70  constraints  that  are  set  forth  under  (11.1)  and  (11.2) 

in  the  preceding  section  after  they  have  first  been  adjusted  in  the  manner 

2 

exhibited  in  Figure  4. 

54.53u1  +  58.98u2  +  38.16u3 

ma X‘  ho  =  86.13V.  +  16.24v  '+  48.21V,  +  49.69v.  +  9vc 
1  2  3  4  5 


subject  to: 


54.53u1  +  58.98u2  +  38.16u3 


-  86.13v.  +  16.24v  +  48.21v_  +  49.69v,  +  9vc 
1  2  3  4  5 


18.82^  +  20.04u2  +  13.77u3 

-  10.44v.  +  4 . 82v„  +  17.13v“+”l7.72v.  +  7v_ 
1  2  3  4  5 


ul’  u2’  U3’  vl*  v2’  V3’  V4’  V5  > 


To  explain  what  is  involved  in  such  adjustments  we  might  first  observe  that  site  1, 
which  appears  in  the  functional  and  in  the  first  constraint,  is  the  same  as  in  (11.2)  since  — 
see  Table  A-6  —  it  was  rated  as  efficient  and  hence  requires  no  adjustment.  That  is,  site  1 
is  already  on  the  PFT  envelope  and  hence  nothing  further  is  required  for  it. 

The  same  is  not  true  for  the  site  70  coefficients  which  appear 
in  the  last  constraint  in  (12).  Comparison  with  (11.2)  will  show  that 
all  of  the  output  coefficients  (in  the  numerator)  have  been  adjusted 


See  also  {  8  ] . 

2 

I.e.,  after  the  observations  have  all  been  brought  on  to  their  respective 
program  envelopes.  We  might  also  note  in  passing,  however,  that  a  still 
different  envelope  may  be  generated  by  affecting  the  efficiency  ratings 
by  reference  to  all  70  constraints  without  first  effecting  any  such 
adjustments.  See  (26). 
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upwards  and  some  of  the  input  coefficients  (in  the  denominator)  have  been  decreased 

in  order  to  bring  site  70  onto  the  NFT  envelope. 

To  see  how  this  is  accomplished  we  first  observe  that  the  effi- 

* 

ciency  of  rating  of  site  number  70  is  hg=  0. 94.  Thus,  to  adjust  its  =  13.69 
component  as  given  in  the  last  row  of  Table  A-4,  we  write 

+  4.31  =  18.82 

where  the  4.31  increment  represents  slack  secured  by  solving  the  linear 
programming  equivalent  of  (11. 2).  * 

Similarly,  we  adjust  the  originally  observed  y^  =  14.19  component 

via 


14.19 

.94 


+  5  =  20.04 


and,  finally,  the  y^  =  12.99  component  via 


12.99 

.94 


13.77 


since  this  output  does  not  involve  any  positive  slack  in  its  optimal  solution. 

The  above  values  represent  the  augmented  outputs  that  site  70 
would  need  to  have  achieved  in  order  to  be  rated  as  efficient  under  NFT. 


This  does  not  end  the  matter,  however,  since  the  program  solution  indicates  that 
6ome  of  the  inputs  should  also  have  been  lower.  In  particular,  the  observed  val¬ 
ue  *  18.21  for  this  site  should  be  replaced  by  =  17.72  and  x,.  =  9  by 
x^  =  7 .  In  other  words  parent  counseling  activities  and  number  of  teachers 
would  also  have  had  to  be  reduced  (in  addition  to  the  indicated  output 

augmentations)  if  site  70  is  to  be  positioned  on  its  portion  of  the  NFT 

2/ 

efficiency  envelope.— 

^Further  details  and  discussion  may  be  found  in  [  9 ]  and  [26]. 

2 

The  point  to  bear  in  mind  is  that  input  dimensions  as  well  as  output  dimensions 
formed  part  of  the  agreed  upon  measures  of  effectiveness  by  which  the  Program  Follow 
Through  experiment  was  to  be  evaluated. 


* 
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Table  A-7  presents  the  results  of  carrying  out  such  calculations 

for  the  inter-envelope  efficiency  ratings.  Statistical  tests  might 

now  be  conducted,  but  inspection  of  Table  A-7  should  suffice  to 

indicate  that  PFT  is  less  efficient  than  NFT.^  This  is  evident  both  in 

the  number  of  DMUs  on^  the  PFT  envelope  which  are  not  efficient  relative 

to  the  inter-envelope  as  well  as  the  fact  that  the  discrepancies  from 

unity  are  generally  larger  than  is  true  for  the  DMUs  that  have  been 

2 

brought  onto  the  NFT  envelope. 


See  [8  1  and  [26]  for  a  detailed  discussion. 

2 

It  is  of  interest  perhaps  to  note  that  the  adjusted  site  70  is  now 
efficient  relative  to  the  inter-envelope  whereas  site  1  is  not. 


<r 


Table  A-7 


Inter-Envelope  Efficiency  Values 


PFT 

Site  # 

* 

h0 

Efficiency  Value 

NFT 

.  Site  # 

* 

h0 

Efficiency  Value 

1 

0.92 

50* 

1.00 

2* 

1.00 

51* 

1.00 

3 

0.94 

52* 

1.00 

4* 

1.00 

53* 

1.00 

5 

0.93 

54* 

1.00 

6* 

1.00 

55 

0.99 

7 

0.99 

56* 

1.00 

8* 

1.00 

57* 

1.00 

9 

0.98 

58* 

1.00 

10 

0.92 

59* 

1.00 

11* 

1.00 

60 

1.00 

12* 

1.00 

61* 

1.00 

13 

0.99 

62* 

1.00 

14 

0.95 

63* 

1.00 

15* 

1.00 

64* 

1.00 

16* 

1.00 

65* 

1.00 

17* 

1.00 

66* 

1.00 

18* 

1.00 

67* 

1.00 

19 

0.99 

.  68 

0.99 

20* 

1.00 

69* 

1.00 

21* 

1.00 

70* 

1.00 

22* 

1.00 

23 

0.99 

24* 

1.00 

25* 

1.00 

26 

0.99 

27* 

1.00 

28* 

1.00 

29 

0.99 

30* 

1.00 

31 

0.99 

32* 

1.00 

33 

0.99 

34 

0.98 

35* 

1.00 

36* 

1.00 

37 

0.94 

38 

0.99 

39* 

1.00 

40 

0.95 

41 

0.99 

42* 

1.00 

43 

0.99 

44* 

1.00 

45 

0.99 

46* 

1.00 

47* 

1.00 

48* 

1.00 

49* 

1.00 

*Denotcs  a  site  with  an  efficiency  value  of  *1 


We  may  now  observe  that  the  efficiency  values  presented  in 


Table  A-7  provide  us  with  the  wanted  estimates  of  output  augmentation 
and/or  resource  conservation.  That  is,  these  ratios  have  the  same  opera¬ 
tional  significance  that  we  earlier  accorded  to  DMU  efficiency  except,  of 
course,  they  now  refer  to  potential  gains  by  removing  the  boundaries  to  efficiency 
that  are  associated  with  the  program —  i**>  the  rules  and  requirements  of  PFT. 

In  the  way  of  still  further  possibilities,  however,  we  might 
also  return  to  Figure  4  and  observe  that  one  portion  of  the  inter- envelope 
is  not  coincident  with  either  PFT  or  NFT.  This  suggests  that  there  may  be 
combinations  of  PFT  and  NFT  which  are  better  than  either.  Such  possibilities 
also  need  to  be  considered  as  part  of  a  PFT  vs.  NFT  decision  and,  of  course, 
as  Figure  4  also  shows, there  may  be  situations  in  which  PFT  is  better  than 
NFT  -  and  so  on. 


CONCLUSION 

These  inferences  concerning  the  relative  efficiencies  of  NFT  and  PFT 
could  be  developed  by  formalized  statistical  tests.  We  do  not  undertake  these 
here,  however,  in  order  to  avoid  the  technical  developments  that  would  be  re¬ 
quired  to  deal  with  properties  such  as  are  involved  in  the  unusual  statistical 
distributions  for  these  measures.  Cf.,  e.g. ,  Figure  3.  The  procedure  associated  with 
DEA  also  involve  certain  novel  features  besides  those  already  indicated. 

In  particular  the  procedures  of  Data  Envelopment  Analysis  reverse  the 
relations  between  statistics  and  substantive  theory  that  are  customary 
in  most  of  the  social  sciences.  The  latter--  i.e.,  customary  usages  -- 


“  44  “ 

generally  apply  statistical  methods  to  the  relations  posited  by  substan¬ 
tive  theory  as  a  condition  precedent  to  their  potential  use.  In  DEA,  on 
the  other  hand,  the  substantive  theory  of  economics — as  formalized  in  the 
definition  of  a  production  function,  say  —  is  first  used  to  adjust  the 
data  prior  to  any  statistical  testing  (e.g.,  along  lines  such  as  were  indicated 
in  connection  with  Figure  4). 

Of  course,  the  point  is  that  we  are  here  seeking  policy  evalua¬ 
tions  rather  than  tests  of  the  substantive  theory  (e.g.,  production  theory) 
we  have  been  employing.  Furthermore,  as  we  indicated  in  our  discussion  of 
"control  predictions"  our  proposed  usage  of  these  methods  depends  on  our 
being  in  a  position  to  enforce  the  assumptions  (such  as  efficient  produc¬ 
tion)  which  we  are  employing.  Given  this  situation,  however,  we  can  then 
say  that  our  proposed  DEA  approach  can  help  to  supply  part  of  the  sus¬ 
tained  pressure  that  is  needed  for  efficiency  in  public  organizations  or 
not-for-profit  entities  when,  the  market  methods  of  competitive  economics 
are  not  a  really  viable  alternative.^ 

This  is  not  to  deny  that  there  are  numerous  activities  in  subdivisions 
of  commercial  enterprises  where  such  alternatives  to  a  direct  application 
of  market  forces  may  be  useful.  Indeed,  in  the  case  of  regulated  indus¬ 
tries  such  as  public  utilities,  etc.,  the  DEA  approach  provides  an  alter¬ 
native  that  is  superior  to  more  standard  econometric  approaches  for 

potential  use  in  the  kinds  of  "efficiency  audits"  that  are  now  being 

2 

ordered  by  certain  public  utility  commissions. 

*For  more  detailed  discussion  along  with  other  possible  developments  and  substl 
tutes,  see  .  [7]. 

2 

See  Ssn  Miguel  [?-]  for  further  discussion. 

.  ,  .x-  ■  fA  -*  ’ 

•  '  t 
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The  distinction  between  program  and  management  efficiency  can  also  be 
adapted  to  the  case  where  technological  rather  than  program  efficiencies  pro¬ 
vide  the  pertinent  alternative  dimensions.  The  comparison  need  not  proceed  by 
pairs  such  as  NFT  vs.  PFT  since  multiple  technologies  or  programs  can  also  be 
accommodated .  — ^ 

We  might  also  say  that  these  alternatives  provide  only  a  beginning  for 
still  further  possibilities.  Distinctions  between  organization  and  technological 
efficiencies  might  be  studied,  for  instance,  in  order  to  evaluate  various  ways 
in  which  different  decision  makers  might  be  organized  under  stipulated  technolo¬ 
gies  and  vice  versa.  See  [6].  How  extensions  to  such  further  new  alternative 
uses  of  DEA  might  be  effected  for  the  still  evolving  activities  involved  in  "com- 
prehensive  audit"  certainly  offers  a  variety  of  exciting  possibilities.  This, 
in  turn,  should  help  to  produce  an  evolution  in  which  subjects  like  managerial 

economics  and  managerial  accounting  will  begin  to  include  significant  public  sector 

2/ 

as  well  as  private  sector  decision-making  problems  on  enhanced  scales.—  Such  an 
alteration  in  educational  courses  and  texts  should  contribute  to  improve  deci¬ 
sion  making  in  public  sector  as  well  as  private  sector  managements  and  thereby 
enhance  the  prospects  for  the  expanded  dimensions  for  accountability  that  are  im¬ 


plied  in  presently  evolving  practices  such  as  corporate  social  reporting  and 

.  .  3/ 

comprehensive  audits.—  Improved  decision  making  and  enhanced  accountability 
could  then  go  hand  in  hand  toward  ameliorating  many  of  the  perplexing  problems 
that  we  seem  to  continually  confront  in  maintaining  a  reasonable  semblance  of 
social  order  and  consensus  while  improving  the  quality  of  life  in  our  ever  more 
rapidly  changing  technologically  oriented  society. 

W  [8]  and  [26]. 

2 

See  [7]  for  further  discussion  and  compare  with  discussion  in  E.  Richardson  [27]. 
3See[31]  and  [14]  and  [IS]. 
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